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The  Honorable  Daniel  K.  Inouye 
Chairman,  Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

Dear  Mr.  Chairman: 

As  requested  by  your  office,  we  reviewed  the  status  of  the  Air  Force’s  pro¬ 
gram  to  acquire  a  training  system  for  its  Special  Operations  Forces.  Your 
Subcommittee  has  been  concerned  about  the  cost  and  eventual  perfor¬ 
mance  of  the  system,  particularly  as  these  aspects  are  affected  by  software 
development.  As  a  result,  the  Senate  Committee  on  Appropriations  denied 
the  fiscal  year  1991  funding  request  for  the  system,  although  it  subse¬ 
quently  agreed  to  restore  the  funding  in  conference.  This  report  follows  up 
on  briefings  given  to  your  office  in  May  and  August  1991. 


Background 


The  Special  Operations  Forces  Aircrew  Training  System  (ats)  is  intended 
to  integrate  classroom,  simulator,  and  in-flight  aircraft  training  for  seven 
different  types  of  aircraft.1  The  ATS  has  a  weapons  system  training  compo¬ 
nent  and  a  mission  rehearsal  component.  The  weapons  system  component 
will  provide  a  full  range  of  aircrew,  mission  qualification,  and  upgrade 
training.  The  mission  rehearsal  component  will  simulate  combat  scenarios 
anywhere  in  the  world,  involving  multiple  types  of  aircraft,  within  48  hours 
of  a  specific  tasking  by  the  National  Command  Authority.2 

The  mission  rehearsal  component  requires  the  development  of  a  com¬ 
puter-generated,  interactive  model  that  simulates  in-flight  images  of  what 
might  exist  in  actual  combat  environments  flown  at  low  altitudes.  It  will 
depict  out-of-the-window  air  and  ground  support  as  well  as  enemy  forces 
and  allow  a  crew  member  to  communicate  or  engage  in  battle,  respectively. 
This  interaction  will  allow  crew  coordination  in  a  simulated  hostile  environ¬ 
ment  performed  under  night  vision,  flying  over  various  terrains,  and  in 
different  weather  conditions.  To  achieve  this  level  of  effort  will  require  the 
development  of  complex  software. 


'The  ATS  will  provide  training  for  transports  (MC-130H  and  MC-130E),  guns  hips  (AC-130H  and 
AC-130U),  tankers  (HC-130),  and  helicopters  (MH-60G  and  MH-53J). 

2The  National  Command  Authority  consists  of  the  President  and  Secretary  of  Defense  or  their  duly  dep¬ 
utized  alternates  or  successors. 
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The  Special  Operations  Forces’  missions  often  require  complete  surprise. 
They  have  only  one  chance  to  accomplish  their  mission.  The  underlying 
requirement  of  mission  rehearsal  is  to  allow,  as  much  as  possible,  aircrews 
to  practice  and  assess  probabilities  of  success,  failure,  detection,  and  sur¬ 
vivability. 

The  ATS  has  a  total  cost  estimated  at  approximately  $400  million  if  all  con¬ 
tract  options  are  exercised.  The  contract  includes  multiple  options,  with 
each  aircraft  being  a  separate  option.  Options  are  exercised  as  funding  is 
appropriated.  We  reviewed  the  two  options  funded  to  date,  which  call  for 
the  development  and  support  of  two  weapons  system  trainers  and  two  mis¬ 
sion  rehearsal  devices  for  the  transport  aircraft.  Work  on  these  options 
started  in  July  1990  and  had  a  contractor-estimated  cost  at  completion  of 
about  $72  dlion.3 
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Even  though  the  Air  Force  has  taken  steps  to  ensure  the  program  is  run 
effectively,  the  contractor  is  experiencing  cost  growth  and  a  schedule 
delay.  To  minimize  risk,  the  Air  Force  used  a  two-phased  acquisition 
approach  involving  contractor  competition  and  also  retained  key  personnel 
to  build  program  continuity.  In  addition,  the  program  office  obtained  co¬ 
ordination  and  cooperation  commitments  from  key  agencies  within  and 
outside  the  Air  Force  at  the  beginning  stages  of  the  contract. 

However,  as  of  July  1 99 1 ,  the  contractor  had  a  cumulative  overrun  of 
29  percent  (about  $6  million)  above  the  $20.5  million  it  had  estimated 
spending  to  that  point  and  was  about  2  months  behind  schedule  on  a  pre¬ 
liminary  software  design  review.  According  to  the  pfogfam  office,  if  the 
overrun  continues,  costs  for  the  first  two  options  bould  rise  to  $94  million, 
or  a  $22  million  increase  above  the  originally  $stfiinat$d  cost  of  $72  miftk>n. 

^  /> 

Also,  the  prime  contractor  rather  than  an  outskte  firm  is  popforminjf  inde¬ 
pendent  verification  and  validation  of  software.  Independent  verifit^tion 
and  validation  is  intended  to  provide  a  high  level  of  assurance  that  the 
software  being  developed  will  work.  In  the  past,  we  have  stated4  that  true 
independence  requires  that  this  effort  not  be  performed  by  the  same  con- 


3 According  to  contract  documents,  the  first  mission  rehearsal  device  is  being  proVicfedfet  no  cost  to  the 
government  by  the  prime  contractor.  It  will  cost  the  contractor  an  estimated  $24  million.  '  -\ 

V  ■ 

4Space  Shuttle:  NASA  Should  Implement  Independent  Oversight  of  Software  Development 
(GAOAMTEC-91-20,  Feb.  22, 199’.;. 
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tractor  that  develops  the  software.  The  Department  of  Defense  (dod) 
Inspector  General  has  taken  a  similar  position.6 


Use  of  Past  Experiences 


Second,  the  program  used  a  contract  incentive  to  enhance  the  performance 
of  and  increase  cooperation  by  the  contractor.  The  prime  contractor  can  be 
awarded  up  to  an  additional  $  1  million  the  first  year  and  over  $1.5  million 
in  each  of  the  following  3  years  under  a  cost-plus-award-fee  payment 
schedule.  The  amounts  are  based  on  a  weighted  performance  evaluation  in 
the  areas  of  cost  control  (30  percent),  management  (25  percent),  quality 
emphasis  (10  percent),  and  technical  engineering  and  support  (35  per¬ 
cent).  The  first  evaluation  period  ended  on  June  30, 1991,  with  the  con¬ 
tractor  receiving  an  overall  rating  of  “very  good,”  which  resulted  in  a 
$700,000  award.  The  Air  Force  elected  to  carry  over  the  remaining 
$300,000  to  the  next  review  period.  This  increases  the  potential  maximum 
award  fee  for  the  second  year  to  $1.8  million. 

Third,  key  Air  Force  personnel  will  be  retained  through  a  critical  design 
review  milestone  in  early  1992,  adding  stability  and  continuity  to  the  pro¬ 
gram.  The  program  manager  was  eligible  for  reassignment  in  January 
1991,  but  his  assignment  was  extended  18  months.  Also,  the  Air  Force  pro¬ 
gram  monitor,  who  has  been  assigned  to  the  program  since  the  beginning 
of  Phase  I,  has  transferred  to  the  U.S.  Special  Operations  Command,  where 
he  is  scheduled  to  monitor  the  program  until  the  total  system  is  deployed. 

Fourth,  Air  Force  program  managers  involved  the  end-users  of  the  aircrew 
training  system  early  on  to  make  certain  that  what  was  to  be  procured 
would  meet  the  users’  needs.  The  issuance  of  the  request  for  proposals  for 


The  ATS  program  has  incorporated  four  successful  elements  from  other  air¬ 
crew  training  system  acquisitions,  which  are  discussed  in  a  recent  Air 
Force  Inspector  General’s  report.6  First,  the  program  used  a  two-phased 
acquisition  approach,  which  the  Air  Force  Inspector  General  reported  has 
produced  solutions  to  aircrew  training  system  problems.  Phase  I  consisted 
of  firm  fixed-price  contracts,  awarded  in  June  1989,  to  three  firms  for 
system  design  studies  to  be  completed  over  an  1 1 -month  period.  These 
studies  supported  the  ATS  Phase  II  competition  for  a  follow-on  contract, 
awarded  in  July  1990,  to  complete  full-scfcle  development,  production,  and 
deployment. 


Acquisition  of  the  C-l  7A  Aircraft  (DOD  IG  Audit  Report  No.  89-059.  Mar.  20, 19891. 

Acquisition  and  Use  of  Training  Devices  (Air  Force  IG  Report  FMI PN  89-641 ,  Dec.  5, 1989  -  Dec.  20, 
1990). 
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the  Phase  I  contract  was  delayed  by  5  months  to  make  certain 
requirements  were  firm  and  agreed  upon,  according  to  the  program  office. 
Also,  the  five  primary  Air  Force  program  offices,  the  Defense  Mapping 
Agency,  the  Defense  Intelligence  Agency,  and  the  National  Security  Agency 
signed  an  agreement  of  cooperation  on  July  20, 1990. 


Cost  Overrun  and 
Schedule  Slippage 


Costs  have  exceeded  budgeted  amounts  since  March  1991  for  the  first  two 
options  of  the  contract.  According  to  the  program  office,  the  overrun  could 
reach  30  percent,  or  $22  million,  bringing  the  total  cost  for  these  options 
to  approximately  $94  million.  The  prime  contractor  has  attributed  the 
majority  of  this  overrun  to  subcontractors’  actual  costs  being  higher  than 
estimated.  Also,  a  lack  of  programming  expertise  in  the  contractor’s  and 
subcontractors’  staffs  required  use  of  more  senior  software  personnel  than 
had  been  anticipated. 


There  has  been  a  2-month  schedule  slippage  because  problems  in  prelimi¬ 
nary  software  design,  development  of  in-class  training  curricula  and  mate¬ 
rials,  and  cockpit  configuration  changes  require  more  work  than 
anticipated.  The  prime  contractor  is  working  with  the  subcontractors  to 
develop  and  implement  a  new  schedule  for  some  tasks  to  meet  all  major 
milestones. 


Independent 
Verification  and 
Validation 


Independent  verification  and  validation  of  software  during  its  development 
is  a  management  tool  used  to  help  ensure  performance,  integrity,  reli¬ 
ability,  safety,  supportabiiity,  and  quality  of  the  final  software  product. 
These  benefits  are  gained  from  the  verification  of  software  requirements, 
designs,  codes,  and  documents.  A  dod  standard7  points  out  that,  to  the 
extent  specified  in  a  contract,  independent  verification  and  validation  is  to 
be  performed  by  a  contractor  or  federal  government  agency  that  is  not 
responsible  for  developing  the  product  or  performing  the  activity  being 
evaluated.  According  to  Air  Force  guidelines,8  independent  verification  and 
validation  should  support  the  program’s  software  development  plan  if, 
among  other  factors,  that  development  is  considered  medium  to  high  risk 
because  of  “technical  reasons”  (e.g.,  complexity,  state-of-the-art,  system 
integration,  and  maturity  of  tools).  The  ATS  program  office  assessed  the 
developmental  risk  as  moderate  to  high,  primarily  in  the  mission  rehearsal 
areas  of  image  generation  and  data  base  development. 


7Military  Standard:  Defense  System  Software  Development  (DOD-STD-216-7  V  Feb.  29,  1988). 

^ftware  Independent  Verification  and  Validation  (IV&V)  (AFSC/AFLC  Pamphlet  800-5,  May  20. 
1988). 
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Recommendation 


Agency  Comments 


However,  the  program  manager  determined  that  the  program  did  not  war¬ 
rant  a  separate  independent  verification  and  validation  effort.  Instead,  he 
elected  to  have  a  unit  within  the  prime  contractor’s  software  quality  assur¬ 
ance  organization  perform  these  functions  and  is  in  the  process  of  pro¬ 
curing  equipment  needed  to  monitor  the  contractor’s  performance.  The 
basis  of  his  decision  was  an  Air  Force  regulation9  that  gives  the  program 
manager  the  option  of  having  independent  verification  and  validation  per¬ 
formed  by  a  prime  contractor  or  a  subcontractor  when  the  unit  to  perform 
this  task  is  autonomous  from  the  software  development  unit. 

Also,  the  program  manager,  citing  Air  Force  guidelines,10  stated  that 
separate  independent  verification  and  validation  was  not  required  because 
the  software  did  not  meet  the  criteria  of  its  failure  causing  death  or  per¬ 
sonnel  iryury,  primary  mission  failure,  or  catastrophic  equipment  loss  or 
damage.  However,  the  program  manager  did  not  mention  the  program’s 
risk  level,  a  criterion  in  the  same  section  of  the  guidelines.  In  addition,  we 
believe  the  nature  of  the  Special  Operations  Forces’  missions  requires  as 
precise  and  accurate  a  mission  rehearsal  as  possible  because,  if  attempted, 
Special  Operation  Forces  get  only  one  chance  to  safely  and  successfully 
accomplish  their  mission.  This  factor  would  also  support  the  need  for  a 
separate  independent  verification  and  validation  effort. 


We  recommend  that  the  Secretary  of  Defense  direct  the  Secretary  of  the 
Air  Force  to  amend  the  Special  Operations  Forces  Aircrew  Training  System 
Program  contract  to  incorporate  an  independent  verification  and  validation 
component  as  specified  in  dod  Standard  2167A.  These  tasks  would  be  con¬ 
ducted  by  a  contractor  or  federal  government  agency  that  is  independent 
of  the  prime  contractor  and  not  responsible  for  developing  the  product  or 
performing  the  activities  being  evaluated. 


dod  officials  agreed  with  our  findings  on  cost  growth  and  schedule  slip¬ 
page  as  they  were  measured  against  the  ATS  contract.  However,  they  stated 
that  they  had  anticipated  and  budgeted  for  such  cost  growth.  Also,  they 
explained  that  the  schedule  slippage  did  not  affect  major  milestone 
completions. 


9Lifecycle  Management  of  Computer  Resources  In  Systems  (AFSC/AFLC  Supplement  1  AFR  800-14, 
Sept,  14, 1987). 

l0Software  Independent  Verification  And  Validation  (IV&V)  (AFSC/AFLC  Pamphlet  800-5,  May  20, 
1988). 
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DOD  officials  also  partially  concurred  with  our  recommendation  and  agreed 
to  reconsider  the  inclusion  of  independent  verification  and  validation  in  the 
ATS  program  to  ensure  software  performance  and  integrity.  DOD’s  written 
comments  are  presented  in  appendix  I. 


o„nnp  nnH  MpthnrinlndV  the  program  office  at  Wright-Patterson  Air  Force  Base,  Dayton, 

^  lU  ^  ^  Ohio;  the  prime  contractor,  Loral  Defense  Systems,  Akron,  Ohio;  the  end 

user,  Headquarters,  Air  Force  Special  Operations  Command,  Hurlburt 
Field,  Florida;  the  U.S.  Special  Operations  Command,  Washington,  D.C., 
Office;  and  the  Offices  of  the  Assistant  Secretary  of  Defense,  Special  Oper¬ 
ations/Low  Intensity  Conflict,  and  the  Secretary  of  the  Air  Force.  Our  work 
consisted  of  interviewing  personnel,  reviewing  contract  documentation, 
and  analyzing  related  reports  and  records. 

We  conducted  our  review  from  January  1991  to  October  1991  in  accor¬ 
dance  with  generally  accepted  government  auditing  standards,  except  that 
we  did  not  independently  verify  cost  and  schedule  data  or  evaluate  the 
basis  for  the  award-fee  rating. 


We  are  sending  copies  of  this  report  to  the  Chairmen,  Senate  Committees 
on  Armed  Services  and  Governmental  Affairs  and  House  Committees  on 
Appropriations,  Armed  Services,  and  Government  Operations;  the  Secre¬ 
taries  of  Defense,  the  Army,  the  Navy,  and  the  Air  Force;  and  the  Director, 
Office  of  Management  and  Budget.  Please  contact  me  at  (202)  275-4268  if 
you  or  your  staff  have  any  questions  concerning  this  report.  The  m^jor 
contributors  to  this  report  are  listed  in  appendix  II. 


Sincerely  yours, 


Ab/cy 

Nancy  R.  Kingsbury  " 

Director 

Air  Force  Issues 
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THE  ASSISTANT  SECRETARY  OF  DEFENSE 
WASHINGTON,  D.C.  20301-2500 


SPECIAL.  OPERATIONS/ 
LOW-INTENSITY  CONFLICT 


Mr.  Frank  C  Conahan 

National  Security  and  International  Affairs  Division 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Conahan: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Accounting  Office 
(GAO)  draft  report,  "DEFENSE  ACQUISITION:  The  Special  Operations  Forces  Aircrew 
Training  System  at  the  One- Year  Mark,"  dated  November  18, 1991  (GAO  Code  392606/OSD 
Case  8893).  The  DoD  partially  concurs  with  the  report. 

The  GAO  correctly  reported  a  contract  cost  overrun.  That  growth  was  anticipated  by 
the  DoD  at  the  time  of  contract  award  and  was  provided  for  in  the  original  program 
acquisition  baseline.  Accordingly,  the  overall  Special  Operations  Forces  Aircrew  Training 
System  program  is  presently  within  the  cost  and  schedule  limits  originally  estimated  and 
established  by  the  DoD  acquisition  baseline. 

With  regard  to  independent  verification  and  validation,  the  DoD  agrees  that  can  be  an 
effective  management  tool.  The  Department  is  satisfied  that  the  Special  Operations  Forces 
Aircrew  Training  System  program  conforms  to  DoD  regulations  and  guidance  and  does  not 
require  independent  verification  and  validation  of  the  software.  The  DoD  is  confident  the 
Special  Operations  Forces  Aircrew  Training  System  program  is  structured  appropriately  to 
deliver  training  systems  that  will  meet  the  demanding  requirements  of  the  U.S.  Special 
Operations  Command.  However,  in  response  to  the  GAO  concerns,  by  January  1992,  the 
Assistant  Secretary  of  Defense  (Special  Operations  and  Low-Intensity  Conflict)  will  direct  U.S. 
Special  Operations  Command  to  reconsider  incorporating  an  independent  verification  and 
validation  requirement  to  enhance  program  execution  and  further  reduce  program  risk. 

The  Department  is  pleased  to  note  the  favorable  GAO  comments  regarding  management 
of  the  program  and  appreciates  the  professional  manner  in  which  the  audit  was  conducted. 
Additional  DoD  comments  on  the  report  findings  and  recommendations  are  provided  in  the 
enclosure.  (Suggested  technical  changes  to  the  report  have  been  provided  separately.)  The 
DoD  appreciates  the  opportunity  to  comment  on  the  GAO  draft  report. 

Sincerely, 


Enclosure 


jJAMtc-  .  L.Jt.liVK  ;i, 

‘'Assistant  Secretary  of  Defense 
(Special  Operations  and  Low- 
tntenaity  Conflict) 
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Appendix  I 

Comment*  From  the  Department  of 
Defense 


Now  on  pp.  1-2. 


GAO  DRAFT  REPORT  -  DATED  NOVEMBER  18. 1991 
(GAO  CODE  392606)  OSD  CASE  8893 

"DEFENSE  ACQUISITION:  THE  SPECIAL  OPERATIONS  FORCES 
AIRCREW  TRAINING  SYSTEM  AT  THE  ONE- YEAR  MARK" 

DEPARTMENT  OF  DEFENSE  COMMENTS 


FINDINGS 


•  FINDING  A  Status  Of  The  Special  Operations  Forces  Aircrew  Training  System.  The 
GAO  observed  that  the  Special  Operations  Forces  Aircrew  Training  System  is  intended 
to  integrate  classroom,  simulator,  and  in-flight  aircraft  training  for  seven  different 
types  of  aircraft.  The  GAO  explained  that  the  System  has  both  a  weapons  system 
training  component  and  a  mission  rehearsal  component.  The  GAO  farther  explained 
that  the  rehearsal  component  will  simulate  combat  scenarios  anywhere  in  the  world,  and 
will  require  the  development  of  a  computer  generated,  interactive  model.  The  GAO 
noted  that,  to  achieve  the  level  of  effort  intended,  the  development  of  complex 
software  will  be  required. 

The  GAO  reported  that  die  Special  Operations  Forces  Aircrew  Training  System  involves 
a  total  contract  cost  of  about  $400  million.  According  to  the  GAO,  the  acquisition 
contract  includes  multiple  options,  with  each  of  the  aircraft  being  a  separate  option. 
The  GAO  reported  that  two  options  have  been  funded  to  date-for  the  development 
and  support  of  two  weapons  system  trainers  and  two  mission  rehearsal  devices  for 
transport  aircraft.  The  GAO  farther  reported  that  work  on  those  two  options 
started  in  July  1990,  with  an  estimated  cost  of  about  $72  million,  (pp.  1-3/GAO 
Draft  Report) 


DoD  Response:  Partially  concur.  The  DoD  agrees,  except  for  the  estimated  program 
cost.  Work  on  the  first  two  Special  Operations  Forces  Aircrew  Training  System 
options  began  in  July  1990,  with  the  signing  of  a  contract  with  Loral  for  $72  million. 
However,  the  DoD  estimated  and  budgeted  for  a  most  probable  cost,  which  exceeds  the 
contract  cost.  The  development  effort  is  funded  under  a  cost-plus  award  fee  contract. 
Under  such  a  contract,  all  allowable  costs  incurred  by  the  contractor,  even  those  over 
and  above  original  contract  value,  must  be  paid  by  the  Government.  Since,  in  the 
estimation  of  the  source  selection  evaluators,  all  competing  contractors  underbid  the 
program,  the  program  office  used  standard  DoD  practices  to  adjust  all  contractor  bids 
to  obtain  a  most  probable  cost  number,  which  was  then  used  in  the  source  selection 
decision. 

Enclosure 
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Appendix  I 

Comment*  From  the  Department  of 
Defense 


NOW  on  pp.  3-4. 


The  DoD  determined  the  Loral  best-and-final  offer  of  $72  million  was  less  than 
necessary  to  complete  the  development  effort.  Accordingly,  the  acquisition  baseline 
established  by  the  DoD  acknowledged  the  need  for  additional  funds  and  committed  the 
required  funds  to  execute  the  first  two  options  of  the  contract. 


S'CIMCJ 


OB:  The  Use  Of  Past  Experience  In  The  Aircrew  Training  System  Acquisition. 
The  GAO  reported  that  the  Special  Operations  Forces  Aircrew  Training  System 
program  has  incorporated  four  successful  elements  from  other  aircrew  training  system 
acquisitions,  as  discussed  in  a  recent  Air  Force  Inspector  General  report  (No.  FM1 PN 
89-64 1 ).  First,  the  GAO  observed  that  the  program  has  used  a  two-phased  acquisition 
approach,  which  approach  the  Air  Force  Inspector  General  determined  had  produced 
solutions  to  aircrew  training  system  problems.  The  GAO  further  reported  that  Phase  I 
of  the  Aircrew  Training  system  consisted  of  firm  fixed  price  contracts  for  system 
design  studies,  awarded  to  three  firms  in  June  1989.  According  to  the  GAO,  those 
studies  supported  the  Phase  II  competition  for  a  follow-on  contract  to  complete 
full-scale  development,  production,  and  deployment,  awarded  in  July  1990. 


The  GAO  noted  that  the  second  element  used  in  the  System  acquisition  was  the  use  of 
a  contract  incentive  to  enhance  performance  of,  and  increase  cooperation  by,  the 
contractor.  The  GAO  reported  that,  under  the  contract,  the  prime  contractor  can  be 
awarded  up  to  an  additional  $1  million  the  first  year,  and  over  $1.3  million  in  each  of 
the  following  three  years.  The  GAO  found  that  for  the  first  evaluation  period,  ending 
June  30,  1991,  the  contractor  was  awarded  $700,000.  and  the  remaining  $300,000 
was  carried  over  to  the  next  review  period.  The  GAO  observed  that,  as  a  result,  the 
potential  maximum  award  fee  for  the  second  year  has  increased  to  $1.8  million. 


The  third  element  repotted  by  the  GAO  is  the  retention  of  key  Air  Force  personnel. 

The  GAO  explained  that  under  that  element,  key  personnel  will  be  retained  through  a 
critical  design  review  milestone  scheduled  for  early  1992 --thus,  adding  stability  and 
continuity  to  the  program. 

The  GAO  reported  that,  under  the  fourth  element,  end  users  of  the  Aircrew  Training 
System  are  to  be  involved  early  on  to  make  certain  users  needs  are  met.  The  GAO 
observed  that,  according  to  Air  Force  officials,  the  request  for  proposals  for  Phase  I 
was  delayed  by  3  months  to  make  certain  requirements  were  firm  and  agreed  upon.  The 
GAO  learned  that,  in  addition,  an  agreement  of  cooperation  among  the  various  DoD 
components  was  signed  on  March  20, 1990.  (pp.  4-6/GAO  Draft  Report) 


DoD  Response:  Concur.  The  DoD  agrees  that  the  use  of  past  program  experience,  the 
comprehensive  management  plan,  and  effective  program  execution  have  been  crucial  to 
the  success  of  this  program. 


FINDING  C:  Cost  Growth  And  Schedule  Slippage  In  The  Aircrew  Training  System 
Acquisition.  The  GAO  found  that,  even  though  the  Air  Force  has  taken  steps  to 
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ensure  the  program  is  run  effectively  (see  Finding  B),  the  program  is  already  experienc¬ 
ing  cost  growth  and  schedule  delays.  The  GAO  found  that,  since  March  1991,  costs 
have  exceeded  budgeted  amounts  for  the  first  two  contract  options.  The  GAO  noted 
that,  as  of  July  1991,  there  was  a  cumulative  overrun  of  29  percent  (or  about 
$6  million)  above  the  amount  budgeted.  The  GAO  reported  that,  according  to  the 
program  office,  the  overrun  could  reach  30  percent  (or  about  $22  million),  bringing 
the  total  cost  for  these  options  to  about  $94  million.  The  GAO  reported  that  the 
prime  contractor  attributed  the  majority  of  the  overrun  to  subcontractor  costs  being 
higher  than  estimated.  The  GAO  indicated  that,  in  addition,  the  lack  of  subcontractor 
and  prime  contractor  programming  expertise  required  the  use  of  more  senior  software 
personnel  than  anticipated. 

The  GAO  also  found  that  the  program  is  2  months  behind  schedule,  due  to 
(1)  problems  in  preliminary  software  design  and  development  of  training  curriculum, 
and  (2)  to  cockpit  configuration  changes  requiring  more  work  than  anticipated. 
According  to  the  GAO,  the  prime  contractor  is  working  with  the  subcontractors  to 
dev<  lop  and  implement  a  new  schedule  to  meet  all  the  original  milestones. 

(p.  3,  pp.  6-7/GAO  Draft  Report) 

DoD  Response:  Partially  concur.  As  discussed  in  the  DoD  response  to  Finding  A,  the 
DoD  budgeted  adequate  funds  to  execute  the  cost-plus  award  fee  contract.  The  GAO 
correctly  reported  the  contractor  cost/scheduie/control  system  reports  reflect 
overruns  to  the  original  underscoped  contract  price.  However,  the  growth  in  the 
latest  contractor  estimate  at  completion  now  accurately  reflects  much  of  the  effort 
the  Government  believed  was  underscoped  at  contract  award  in  July  1990.  Although 
the  contract  price  has  been  exceeded  with  allowable,  and  mostly  expected  costs,  the 
budgeted  program  cost  at  completion,  which  is  the  DoD  acquisition  baseline,  remains 
unchanged.  The  program  office  closely  tracks  contract  costs  on  a  monthly  basis  with 
analysis  of  contract  cost  performance  reports,  and  looks  for  variances  which  could  lead 
to  a  baseline  breach.  Any  breach  to  the  acquisition  program  baseline  would  be 
reported  by  the  Air  Force  using  established  DoD  procedures. 

While  the  contractor  documents  do  indicate  a  two-month  slip  on  one  effort,  the 
overall  program  remains  on  schedule.  At  the  advice  of  the  Government  program 
engineering  office,  the  Government  purposely  slipped  an  electronic  combat  environment 
software  development  effort  to  clarify  key  prime-item  development  specification  issues 
with  the  using  comand  prior  to  the  contractor  proceeding  with  that  portion  of 
software  development  work.  The  contractor  cost  accounting  system  correctly  showed 
that  u  work  not  being  accomplished.  The  two-month  delay  in  the  sub-system 
software  development  effort  has  caused  no  program  slip  because  the  development 
effort  is  not  on  the  critical  path.  Minor  rescheduling  efforts  of  this  nature  occur 
regularly  throughout  any  development  program  and,  in  this  case,  are  being  appropri¬ 
ately  used  by  the  Government  and  contractor  managers  to  ensure  the  program  meets 
the  user  needs  and  also  stays  within  cost  and  schedule. 
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The  problems  cited  in  developing  the  in-class  training  curriculum  and  in  developing 
cockpit  configuration  changes  have  not  caused  any  overall  program  schedule  slips  and 
are  unrelated  to  the  preliminary  software  design. 

•  FINDING  D;  Approach  To  Independent  Verification  And  Validation  For, The  Aircrew 
Training  System.  The  GAO  observed  that  independent  verification  and  validation  of 
software  during  development  is  a  management  tool  to  help  ensure  performance, 
integrity,  reliability,  safety,  supportability,  and  quality  of  the  final  software  product. 
The  GAO  noted  that,  according  to  Air  Force  guiddines,  independent  verification  and 
validation  should  support  the  software  development  plan  for  a  program  if,  among 
other  factors,  the  development  is  considered  medium  to  high  risk  because  of  technical 
reasons.  The  GAO  noted  that,  in  the  case  of  the  Aircrew  Training  System,  the 
program  office  assessed  the  development  risk  as  medium  to  high-primarily  concerning 
areas  in  mission  rehearsal. 

The  GAO  found,  however,  that  the  program  manager  determined  the  Aircrew  Training 
System  did  not  warrant  a  separate  independent  verification  and  validation  effoit.  The 
GAO  reported  that  instead,  a  unit  within  the  software  quality  assurance  organization 
of  the  prime  contractor  was  to  perform  those  functions.  According  to  the  GAO,  the 
basis  for  that  decision  was  an  Air  Force  regulation  (Supplement  1/  Air  Force  Regula¬ 
tion  800- 14),  which  gives  the  program  manager  the  option  of  having  independent 
verification  and  validation  performed  by  a  prime  contractor  or  subcontractor.  The 
GAO  asserted,  however,  that  DoD  Standard  2 167 A  indicates  that  independent 
verification  and  validation  should  be  performed  by  a  contractor  or  Federal  agency  not 
responsible  for  developing  the  product  or  performing  the  activity  being  evaluated- 

The  GAO  further  repotted  that  the  program  manager  also  cited  an  Air  Force  pamphlet 
(Pamphlet  800-5)  as  a  reason  independent  verification  and  validation  was  not  required, 
since  the  program  did  not  meet  the  criteria  of  software  problems  causing  ( 1 )  personnel 
injury,  (2)  primary  mission  failure,  or  (3)  catastrophic  equipment  loss  or  damage~as 
cited  in  the  pamphlet.  The  GAO  pointed  out,  however,  that  the  program  manager  did 
not  mention  the  risk  level  of  the  program- a  criterion  also  contained  in  the  same 
pamphlet.  The  GAO  cited  its  February  1991  report  on  software  development  for  the 
Space  Shuttle  (GAO/IMTEC-9I-20),  in  which  the  GAO  stated  that  tnie  independence 
requires  that  the  effort  not  be  performed  by  the  same  contractor  that  develops  the 
software.  The  GAO  reported  that  the  DoD  Inspector  General  took  a  similar  position 
in  a  March  1989  report  (DoD  Inspector  General  Report  89-059)  concerning  acquisition 
of  the  C-17A.  The  GAO  concluded  that  the  nature  of  the  Special  Operations  Forces 
missions  requires  as  precise  and  accurate  a  mission  rehearsal  as  possible,  since  the 
Special  Operation  Forces  get  only  one  chance  to  accomplish  their  mission  safely  and 
successfully,  (p.  4,  pp.  7-8/GAO  Draft  Report) 


DoD  Response:  Partially  concur.  The  DoD  agrees  that  independent  verification  and 
validation  is  an  effective  management  tool  when  circumstances  dictate.  Although  the 
Department  is  satisfied  that  the  Aircrew  Training  System  program  office  properly 
applied  DoD  directives  and  Air  Force  guidance  in  determining  that  no  independent 
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verification  and  validation  of  software  is  required  for  the  Aircrew  Training  System 
program,  because  of  GAO  concern,  incorporation  of  independent  verification  and 
validation  will  be  reconsidered  to  ensure  software  performance  and  integrity.  (Also  see 
the  DoD  responses  to  Recommendations  1  and  2.) 

The  determination  not  to  require  independent  verification  and  validation  was  based  in 
part  on  criteria  in  Chapter  2  of  Air  Force  Systems  Command/Air  Force  Logistics 
Command  Pamphlet  800-5,  Software  Independent  Verification  and  Validation.  Since 
none  of  the  criteria  (including  risk)  apply  to  this  program,  the  program  office 
determined  that  independent  verification  and  validation  was  not  required  or  cost 
effective.  In  particular,  although  the  overall  Special  Operations  Forces  Aircrew 
Training  System  technical  development  effort  is  medium  to  high  risk,  as  the  GAO 
acknowledged  it  is  due  to  the  mission  rehearsal  aspects  of  image  generation  and  data 
base  development,  not  software  development.  It  is  critical  to  distinguish  between 
overall  system  technical  risk  versus  software  development  risk. 

Applicable  DoD  Standard  2 167 A  does  not  mandate  independent  verification  and 
validation;  it  requires  contractor  interface  with  the  software  independent  verification 
and  validation  agent,  if  independent  verification  and  validation  is  specified  in  the 
contract.  The  Aircrew  Training  System  contract  contains  no  provision  for  independent 
verification  and  validation. 

The  existing  software  development  program  was  determined  not  to  have  inherent  risks 
that  would  warrant  the  unprogrammed  expense  of  $3-4  million  for  a  separate  software 
independent  verification  and  validation.  Adequate  oversight  of  the  software  develop¬ 
ment  effort  is  in  place  to  protect  the  interests  of  the  DoD  and  to  ensure  successful 
software  development.  The  independent  software  quality  assurance  organization  of  the 
prime  contractor  inspects,  audits,  and  samples  process  conformance,  in  accordance  with 
the  provisions  of  the  contract  and  in  compliance  with  the  approved  software  quality 
assurance  plan.  In  addition,  the  use  of  networked,  compatible  software  engineering 
environments  at  all  the  contractor  locations  and  at  Wright-Patteison  Air  Force  Base 
provides  the  Air  Force  on-line  access  to  monitor  and  review  the  contractor  results  as 
the  software  is  being  developed.  Furthermore,  the  program  office  routinely  consults 
the  Software  Engineering  Institute  at  Carnegie  Melon  University  on  the  latest 
techniques  and  methods  with  respect  to  simulator  software.  The  Software  Engineering 
Institute  collaborates  with  program  personnel  during  design  reviews,  by  monthly 
software  working  group  meetings,  and  by  telephone,  as  needed.  As  a  final  risk 
management  measure,  the  program  office  and  Loral  track  (1)  software  lines  of  code, 

(2)  memory  utilization,  (3)  software  manpower,  (4)  software  action  items  and 
(S)  software  cost/schedule/performance  with  numerous  management  indicators.  The 
program  is  currently  meeting  or  exceeding  goals  in  all  five  areas. 

The  Department  originally  directed  the  funding  and  implementation  of  this  program  in 
order  to  provide  a  responsive  mission  rehearsal  cabability  for  nationally  important 
missions.  While  the  GAO  correctly  notes  that  Special  Operations  Forces  missions 
require  as  precise  and  accurate  a  mission  rehearsal  as  possible,  this  is  principally  a 
testing  and  fidelity  issue  for  the  mission  rehearsal  devices.  The  DoD  has  planned 
extensive  testing  to  verify  system  performance.  Incrementally  throughout  the  the 
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Now  on  p.  6. 


development  process,  fully  qualified  crew  members  will  continue  to  support  contractor 
testing  to  verify  the  accuracy  and  fidelity  of  each  aircraft  training  device  prior  to  final 
acceptance  testing  by  the  Air  Force.  For  final  acceptance,  the  weapon  system  trainers 
are  being  tested  to  Federal  Aviation  Administration  Phase  II  certification  standards, 
which  are  extremely  rigorous.  The  fidelity  of  the  electronic  combat  environment  will 
be  validated  independently  by  the  Air  Force  Electronic  Warfare  Center.  An  operational 
test  of  the  Special  Operations  Forces  Aircrew  Training  System  will  be  planned  and 
conducted  by  the  U.S.  Air  Force  Airlift  Center,  Military  Airlift  Command,  and 
monitored  by  the  Air  Force  Operational  Test  and  Evaluation  Center. 


***** 

RECOMMENDATIONS 


•  RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary 

of  Defense  direct  the  Secretary  of  the  Air  Force  to  incorporate  an  independent 
verification  and  validation  requirement  in  the  Special  Operations  Forces  Aircrew 
Training  System  Program,  (p.  9/GAO  Draft  Report) 

DoP  response:  Partially  concur.  As  discussed  in  the  the  DoD  response  to  Finding  D, 
independent  verification  and  validation  is  not  required  for  the  Special  Operations  Forces 
Aircrew  Training  System  program.  Software  development  is  being  accomplished  in  full 
compliance  with  applicable  DoD  policies  and  procedures.  However,  due  to  the  GAO 
concerns,  by  January  1992,  the  Assistant  Secretary  of  Defense  (Special  Operations  and 
Low  Intensity  Conflict)  will  direct  the  U.S.  Special  Operations  Command  to  reconsider 
incorporating  an  independent  verification  and  validation  requirement  in  the  Aircrew 
Training  System  program  to  further  reduce  program  risk. 

•  RECOMMENDATION  2:  The  GAO  further  recommended  that  the  Secretary  of  Defense 
direct  the  Secretary  of  the  Air  Force  to  ensure  those  [independent  verification  and 
validation]  tasks  be  conducted  by  a  contractor  or  Federal  agency  that  is  independent  of 
the  prime  contractor,  and  not  responsible  for  developing  the  product  or  performing 
the  activities  being  evaluated,  (p.  9/GAO  Draft  Report) 

DoD  response:  Partially  concur.  As  discussed  in  the  DoD  response  to  Finding  D, 
independent  verification  and  validation  is  not  required  for  the  Special  Operations  Forces 
Aircrew  Training  System  program.  Software  development  is  being  accomplished  in  full 
compliance  with  applicable  DoD  policies  and  procedures.  The  DoD  is  confident  the 
Special  Operations  Forces  Aircrew  Training  System  program  is  structured  appropriately 
and  the  delivered  training  systems  will  meet  the  demanding  U.S.  Special  Operations 
Command  requirements.  However,  due  to  the  GAO  concerns,  by  January  1992,  the 
Assistant  Secretary  of  Defense  (Special  Operations  and  Low  Intensity  Conflict)  will 
direct  the  U.S.  Special  Operations  Command  to  reconsider  incorporating  an  independent 
verification  and  validation  requirement  in  the  Aircrew  Training  System  program  to 
further  reduce  program  risk. 
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The  first  copy  of  each  GAO  repot  i  Is  ft  ee.  Additional  copies  axe  $2 
each.  Orders  should  be  sent  to  the  following  address  accompanied 
by  a  check  or  money  order  made  out  to  the  Superitm  ndent  of  Docu¬ 
ments,  when  necessary.  Orders  for  100  or  more  copie  to  IxT mailed 
to  a  single  address  are  discounted  25  percent. 

U.S.  General.Accountii  g  Office 
JP.O.  Box  6015 
Gaithersburg,  MD  20H77 

Orders  may  also  be  pieced  by  calling  (202)  275-6241. 


